3D Reconstruction Workflow¶

After bowl and voltage calibration we are ready to calculate the 3d reconstruction. In this workflow we calculate the reconstructed x,y,z and then plot the 3d, heatmap, projection plots and mass-to-charge histogram.

In [1]:
# Activate intractive functionality of matplotlib
%matplotlib ipympl
# Activate auto reload
%load_ext autoreload
%autoreload 2
%reload_ext autoreload
# import libraries
import os
import pandas as pd
from ipywidgets import fixed, interact_manual, widgets
from ipywidgets import VBox
from IPython.display import clear_output
from IPython.display import display
from ipywidgets import Output
import matplotlib.colors as mcolors
import warnings
# Ignore all warnings
warnings.filterwarnings("ignore")
# to active the intractive of plotlu to the save html file
import plotly.io as pio
pio.renderers.default = 'jupyterlab'

# Local module and scripts
from pyccapt.calibration.calibration_tools import share_variables, ion_selection
from pyccapt.calibration.calibration_tools import widgets as wd
from pyccapt.calibration.data_tools import data_tools, dataset_path_qt
from pyccapt.calibration.reconstructions import reconstruction
from pyccapt.calibration.calibration_tools import mc_plot

By clicking on the button below, you can select the dataset file you want to use. The dataset file can be in various formats, including HDF5, EPOS, POS, ATO, and CSV.

In [2]:
button = widgets.Button(
    description='load dataset',
)
@button.on_click
def open_file_on_click(b):
    """
    Event handler for button click event.
    Prompts the user to select a dataset file and stores the selected file path in the global variable dataset_path.
    """
    global dataset_path
    dataset_path = dataset_path_qt.gui_fname().decode('ASCII')
button
Out[2]:
In case of recieving the error about pytable library, you have to install the pytables library with conda command. to do that you can open a new cell and copy the line below in it. Then just run it like other cells. The pytables library will be innstalled.

!conda install --yes --prefix {sys.prefix} pytables

If you ranged the dataset, you can select the range file by clicking on the button below. The range file can be created with "ion selection and ranging" workflow. If you do not have a range file, you can skip this step.
In [3]:
button_r = widgets.Button(
    description='load range dataset',
)
@button_r.on_click
def open_file_on_click_r(b):
    """
    Event handler for button click event.
    Prompts the user to select a range file and stores the selected file path in the global variable range_path.
    """
    global range_path
    range_path = dataset_path_qt.gui_fname().decode('ASCII')
button_r
Out[3]:
From the dropdown lists below, you can select the instrument specifications of the dataset. The TDC model and flight path distance are required for the reconstruction.
In [4]:
tdc, pulse_mode, flightPathLength, t0_d, max_mc, det_diam = wd.dataset_instrument_specification_selection()
display(tdc, flightPathLength)
In [5]:
# exctract needed data from Pandas data frame as an numpy array
# create an instance of the Variables opject
variables = share_variables.Variables()
variables.pulse_mode = pulse_mode
dataset_main_path = os.path.dirname(dataset_path)
dataset_main_path = os.path.dirname(dataset_main_path)
dataset_name_with_extention = os.path.basename(dataset_path)
variables.dataset_name = os.path.splitext(dataset_name_with_extention)[0]
variables.result_data_path = dataset_main_path + '/reconstruction/'
variables.result_data_name = variables.dataset_name
variables.result_path = dataset_main_path + '/reconstruction/'

if not os.path.isdir(variables.result_path):
    os.makedirs(variables.result_path, mode=0o777, exist_ok=True)
    
# Create data farame out of hdf5 file dataset
data = data_tools.load_data(dataset_path, tdc.value, mode='processed')
# extract data from the path and create the Variable object
data_tools.extract_data(data, variables, flightPathLength.value, max_mc.value)

The maximum time of flight: 5010
In [6]:
data
Out[6]:
x (nm) y (nm) z (nm) mc_c (Da) mc (Da) high_voltage (V) pulse start_counter t_c (ns) t (ns) x_det (cm) y_det (cm) pulse_pi ion_pp
0 0.0 0.0 0.0 13.450285 14.136714 5019.720215 1003.943970 3495 412.899765 446.853577 2.964898 -0.169796 0 0
1 0.0 0.0 0.0 26.983236 29.616535 5019.720215 1003.943970 3565 569.722887 616.451904 -1.936327 0.088163 70 2
2 0.0 0.0 0.0 27.722095 30.456534 5019.720215 1003.943970 4103 576.981893 623.172729 -1.648980 0.672653 538 1
3 0.0 0.0 0.0 26.956814 29.550233 5019.720215 1003.943970 4134 569.461476 616.253052 -1.975510 -0.218776 31 1
4 0.0 0.0 0.0 27.029927 28.966265 5019.720215 1003.943970 4205 570.184518 605.431091 1.296327 -0.173061 71 1
... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
10191150 0.0 0.0 0.0 128.577300 141.314648 6347.270020 1269.453979 8768 1201.675435 1154.633423 -0.734694 -1.577143 403 1
10191151 0.0 0.0 0.0 27.145753 29.461296 6347.270020 1269.453979 8799 571.327970 548.825195 0.408163 1.508571 31 1
10191152 0.0 0.0 0.0 27.070999 29.600126 6347.270020 1269.453979 9443 570.590274 547.803345 -1.165714 0.097959 485 1
10191153 0.0 0.0 0.0 26.987780 29.719453 6347.270020 1269.453979 9771 569.767836 555.038513 -1.645714 1.296327 328 1
10191154 0.0 0.0 0.0 27.160412 29.823040 6347.270020 1269.453979 10327 571.472518 556.574707 -0.982857 1.933061 556 1

10191155 rows × 14 columns

Cell below shows apply the loaded range file and show the range dataset with related colors. If you do not have a range file, the code will create a range file with the name of "unranged" and color of black.

In [7]:
if 'range_path' in locals():
    variables.range_data = data_tools.read_hdf5_through_pandas(range_path)

else:
    variables.range_data = pd.DataFrame({"ion": ['unranged'], "mass": [0], "mc": [0], "mass": [0], "mc_low": min(variables.mc_calib), "mc_up": max(variables.mc_calib), "color": ['#000000'], "element": ['unranged'],
                                        "complex": [0], "isotop": [0], "charge": [0]})
display(variables.range_data.style.applymap(ion_selection.display_color, subset=['color']))
  ion mass mc mc_low mc_up color element complex isotop charge
0 unranged 0 0 0.042695 399.999838 #000000 unranged 0 0 0
You can change the color of the range by clicking on the "change color" button.
In [8]:
change_color = widgets.Button(
    description='change color',
)
color_picker = widgets.ColorPicker(description='Select a color:')
row_index = widgets.IntText(value=0, description='index row:')
@change_color.on_click
def change_color_m(b,):
    with out:
        selected_color = mcolors.to_hex(color_picker.value)
        variables.range_data.at[row_index.value, 'color'] = selected_color
        clear_output(True)
        display(variables.range_data.style.applymap(ion_selection.display_color, subset=['color']))
        
# Create a container for the widgets
container = widgets.VBox([color_picker, row_index, change_color])
out = Output()
# Display the widgets
display(container, out)

The calibrated mass-to-charge ratio that is calculated with previous workflow will be shown below. You can select the range of the data you want to plot. There is also possibility to activate the peak finding tool to show the peak values.

In [12]:
interact_manual(
        mc_plot.hist_plot,
        variables=fixed(variables),
        bin_size=widgets.FloatText(value=0.1),
        log=widgets.Dropdown(options=[('True', True), ('False', False)]),
        target=widgets.Dropdown(options=[('mc_c', 'mc_c'), ('tof_c', 'tof_c'), ('mc', 'mc'), ('tof', 'tof')]),
        mode=widgets.Dropdown(options=[('normal', 'normal'), ('normalized', 'normalized')]),
        prominence=widgets.IntText(value=100),
        distance=widgets.IntText(value=100),
        percent=widgets.IntText(value=50),
        selector=fixed('None'),
        figname=widgets.Text(value='hist'),
        lim=widgets.IntText(value=variables.max_mc),
        peaks_find_plot=widgets.Dropdown(options=[('True', True), ('False', False)]),
        peaks_find=fixed(True),
        range_plot=fixed(False),
        plot_ranged_ions=fixed(False),
        ranging_mode=fixed(False),
        selected_area_specially=fixed(False),
        selected_area_temporally=fixed(False),
        save_fig=widgets.Dropdown(options=[('True', True), ('False', False)]),
        print_info=fixed(True),
        figure_size=fixed((9, 5)));

You have to select the main element in your sample from the from dropdown below.

In [13]:
element_selected = wd.density_field_selection()
display(element_selected)
In [14]:
avg_dens = element_selected.value[2]
field_evap = element_selected.value[3]

By specifying the parameters such as kf, icf, and detector efficiency, you can reconstruct your sample and generate a final 3D visualization. The 'element_percentage' allows you to select a specific subset of ions to plot, helping to avoid system overload. Additionally, you have the option to enable 'rotary_fig_save' as a boolean to save the 3D plot in a rotary format.

In [19]:
if variables.range_data.empty:
    element_percentage = str([0.01])
else:
    element_percentage = [0.01] * len(variables.range_data['element'].tolist())
    element_percentage = str(element_percentage)
    

interact_manual(reconstruction.x_y_z_calculation_and_plot, variables=fixed(variables), element_percentage=widgets.Textarea(value=element_percentage),
                kf=widgets.FloatText(value=4), det_eff=widgets.FloatText(value=0.7), icf=widgets.FloatText(value=1.4),
                field_evap=widgets.FloatText(value=field_evap),
               avg_dens=widgets.FloatText(value=avg_dens), flight_path_length=fixed(flightPathLength.value),
               rotary_fig_save=widgets.Dropdown(options=[('True', True), ('False', False)], value=False), 
                selected_are=fixed(False),  mode=widgets.Dropdown(options=[('Gault', 'Gault'), ('Bas', 'Bas')]),
               opacity=widgets.FloatText(value=0.5, min=0, max=1, step=0.1), figname=fixed('3d'), save=widgets.Dropdown(options=[('True', True), ('False', False)]));
In [20]:
data['x (nm)'] = variables.x
data['y (nm)'] = variables.y
data['z (nm)'] = variables.z
data
Out[20]:
x (nm) y (nm) z (nm) mc_c (Da) mc (Da) high_voltage (V) pulse start_counter t_c (ns) t (ns) x_det (cm) y_det (cm) pulse_pi ion_pp
0 23.730815 -1.359033 4.425380 13.450285 14.136714 5019.720215 1003.943970 3495 412.899765 446.853577 2.964898 -0.169796 0 0
1 -15.933121 0.725454 1.954712 26.983236 29.616535 5019.720215 1003.943970 3565 569.722887 616.451904 -1.936327 0.088163 70 2
2 -13.613284 5.553142 1.657164 27.722095 30.456534 5019.720215 1003.943970 4103 576.981893 623.172729 -1.648980 0.672653 538 1
3 -16.237998 -1.798258 2.052436 26.956814 29.550233 5019.720215 1003.943970 4134 569.461476 616.253052 -1.975510 -0.218776 31 1
4 10.790760 -1.440580 0.903395 27.029927 28.966265 5019.720215 1003.943970 4205 570.184518 605.431091 1.296327 -0.173061 71 1
... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
10191150 -7.675634 -16.477028 57.298074 128.577300 141.314648 6347.270020 1269.453979 8768 1201.675435 1154.633423 -0.734694 -1.577143 403 1
10191151 4.278334 15.812721 56.918286 27.145753 29.461296 6347.270020 1269.453979 8799 571.327970 548.825195 0.408163 1.508571 31 1
10191152 -12.293745 1.033088 56.212232 27.070999 29.600126 6347.270020 1269.453979 9443 570.590274 547.803345 -1.165714 0.097959 485 1
10191153 -17.062977 13.440480 58.169980 26.987780 29.719453 6347.270020 1269.453979 9771 569.767836 555.038513 -1.645714 1.296327 328 1
10191154 -10.172648 20.007334 58.368539 27.160412 29.823040 6347.270020 1269.453979 10327 571.472518 556.574707 -0.982857 1.933061 556 1

10191155 rows × 14 columns

In [29]:
plot_3d_button = widgets.Button(
    description='plot 3D',
)
plot_heatmap_button = widgets.Button(
    description='plot heatmap',
)
plot_mc_button = widgets.Button(
    description='plot mc',
)
plot_projection_button = widgets.Button(
    description='plot projection',
)

clear_button = widgets.Button(
    description='Clear plots',
)

figname_3d=widgets.Text(value='3d_plot', description='fig name')
selected_area_p3=widgets.Dropdown(options=[('False', False), ('True', True)], description='Selected area')
rotary_fig_save_p3=widgets.Dropdown(options=[('True', True), ('False', False)], description='Rotary save')
element_percentage_p3=widgets.Textarea(value=element_percentage, description='Element percentage')
@plot_3d_button.on_click
def plot_3d(b):
    with out:
        if selected_area_p3.value:
            variables.selected_z1 = variables.selected_y1
            variables.selected_z2 = variables.selected_y2
            variables.selected_y1 = variables.selected_x1
            variables.selected_y2 = variables.selected_x2
            print('Min x (nm):', variables.selected_x1, 'Max x (nm):', variables.selected_x2)
            print('Min y (nm):', variables.selected_y1, 'Max y (nm):', variables.selected_y2)
            print('Min z (nm):', variables.selected_z1, 'Max z (nm):', variables.selected_z2)
        opacity = 1
        reconstruction.reconstruction_plot(variables, element_percentage_p3.value, opacity, rotary_fig_save_p3.value,  figname_3d.value, save=True,
                                           selected_area_specially=selected_area_p3.value, selected_area_temporally=selected_area_p3.value, ions_individually_plots=False)

selected_area_ph=widgets.Dropdown(options=[('False', False), ('True', True)], description='Selected area')
element_percentage_ph=widgets.Textarea(value=element_percentage, description='Element percentage')
figname_heatmap=widgets.Text(value='heatmap', description='fig name')
@plot_heatmap_button.on_click
def plot_heatmap(b, variables=variables):
    with out:
        if selected_area_ph.value:
            variables.selected_z1 = variables.selected_y1
            variables.selected_z2 = variables.selected_y2
            variables.selected_y1 = variables.selected_x1
            variables.selected_y2 = variables.selected_x2
            print('Min x (nm):', variables.selected_x1, 'Max x (nm):', variables.selected_x2)
            print('Min y (nm):', variables.selected_y1, 'Max y (nm):', variables.selected_y2)
            print('Min z (nm):', variables.selected_z1, 'Max z (nm):', variables.selected_z2)

        reconstruction.heatmap(variables, selected_area_ph.value, selected_area_ph.value, element_percentage_ph.value, figure_name='heat_map', figure_sie=(5,5), save=True)

selected_area_pm=widgets.Dropdown(options=[('False', False), ('True', True)], description='Selected area')
peak_find_plot=widgets.Dropdown(options=[('True', True), ('False', False)], description='peak find')
rangging=widgets.Dropdown(options=[('False', False), ('True', True)], description='rangging')
range_file_exist_pm=widgets.Dropdown(options=[('True', True), ('False', False)], description='Range exist')
bin_size_pm = widgets.FloatText(value=0.1, description='Bins size')
lim_mc_pm = widgets.IntText(value=150, description='Limit mc')
prominence=widgets.IntText(value=50, description='peak prominance:')
distance=widgets.IntText(value=50, description='peak distance:')
hist_color_range=widgets.Dropdown(options=[('True', True), ('False', False)], description='hist color')
figname_mc=widgets.Text(value='mc', description='fig name')
@plot_mc_button.on_click
def plot_mc(b,):
    with out:
        if selected_area_pm.value:
            variables.selected_z1 = variables.selected_y1
            variables.selected_z2 = variables.selected_y2
            variables.selected_y1 = variables.selected_x1
            variables.selected_y2 = variables.selected_x2
            print('Min x (nm):', variables.selected_x1, 'Max x (nm):', variables.selected_x2)
            print('Min y (nm):', variables.selected_y1, 'Max y (nm):', variables.selected_y2)
            print('Min z (nm):', variables.selected_z1, 'Max z (nm):', variables.selected_z2)

        mc_plot.hist_plot(variables, bin_size_pm.value, log=True, target='mc', mode='normal', prominence=prominence.value, distance=distance.value, percent=50, selector='None', figname=figname_mc.value, lim=lim_mc_pm.value,
                          peaks_find_plot=peak_find_plot.value, peaks_find=True, range_plot=rangging.value, plot_ranged_ions=False, ranging_mode=False, selected_area_specially=selected_area_pm.value,
                          selected_area_temporally=False, save_fig=True, print_info=True, figure_size=(9, 5))

element_percentage_pp=widgets.Textarea(value=element_percentage, description='Element percentage')
selected_area_pp=widgets.Dropdown(options=[('False', False), ('True', True)], description='Selected area')
x_or_y_pp=widgets.Dropdown(options=['x', 'y'], value='x', description='X or Y')
figname_p = widgets.Text(value='projection', description='fig name')
@plot_projection_button.on_click
def plot_projection(b,):
    with out:
        if selected_area_pp.value:
            variables.selected_z1 = variables.selected_y1
            variables.selected_z2 = variables.selected_y2
            variables.selected_y1 = variables.selected_x1
            variables.selected_y2 = variables.selected_x2
            print('Min x (nm):', variables.selected_x1, 'Max x (nm):', variables.selected_x2)
            print('Min y (nm):', variables.selected_y1, 'Max y (nm):', variables.selected_y2)
            print('Min z (nm):', variables.selected_z1, 'Max z (nm):', variables.selected_z2)

        reconstruction.projection(variables, element_percentage_pp.value, selected_area_pp.value, selected_area_pp.value, x_or_y_pp.value, figname_p.value,
                                 figure_size=(5,5), save=True)

@clear_button.on_click
def clear(b,):
    with out:
        clear_output(True)
        print('')


tab1 = VBox(children=[selected_area_pp, x_or_y_pp, element_percentage_pp, figname_p, plot_projection_button, clear_button])
tab2 = VBox(children=[selected_area_p3, rotary_fig_save_p3, element_percentage_p3, figname_3d, plot_3d_button, clear_button])
tab3 = VBox(children=[selected_area_pm, bin_size_pm, prominence, distance, lim_mc_pm, peak_find_plot, rangging, figname_mc, plot_mc_button, clear_button])
tab4 = VBox(children=[selected_area_ph, element_percentage_ph, figname_heatmap, plot_heatmap_button, clear_button])

tab = widgets.Tab(children=[tab1, tab2, tab3, tab4])
tab.set_title(0, 'projection')
tab.set_title(1, '3d plot')
tab.set_title(2, 'mc plot')
tab.set_title(3, 'heatmap plot')

out = Output()

Here you can plot the 3d, heatmap, projection plots and mass-to-charge histogram. You can also save the plots by clicking on the "save plots" button. It is also possible to select a specific area of projection plot and then plot the 3d, heatmap, projection plots and mass-to-charge histogram of the selected area. You can also save the plots of the selected area by clicking on the "save plots" button.

In [30]:
display(VBox(children=[tab]))
display(out)

Save the cropped dataset. You can specify te output format from list below. The output formats are HDF5, EPOS, POS, ATO, and CSV. The output file will be saved in the same directory as the original dataset file in a new directory nammed load_crop.

In [31]:
interact_manual(data_tools.save_data, data=fixed(data), variables=fixed(variables),
                hdf=widgets.Dropdown(options=[('True', True), ('False', False)]),
                epos=widgets.Dropdown(options=[('False', False), ('True', True)]), 
                pos=widgets.Dropdown(options=[('False', False), ('True', True)]), 
                ato_6v=widgets.Dropdown(options=[('False', False), ('True', True)]), 
                csv=widgets.Dropdown(options=[('False', False), ('True', True)]));